NOTE this page has been prepared in anticipation of the approval of the Reference Releases. It does not reflect the current working procedure of OMA.
the current procedure is at http://member.openmobilealliance.org/ftp/rel/gen_info/consistency.shtml
Updated on 2004-04-28, clarifying text added on:
Updated on 2004-07-28, changes made to:
Updated on 2005-03-04, changes made to:
Updated on 2005-09-09, changes made to:
Updated on 2005-12-12 to to add text for ETR RR
Updated on 2006-05-11 to add Reference Releases (OMA-REL-2006-0053R01-CR_CONRProcedures_RefReleases )
Updated 0n 2006-05-23 to remove WGs as reviewers (OMA-REL-2006-0071R02-Removing-WGs-as-reviewers-in-CONR)
From an early point of time, the Release Planning and Management committee will contact a Working Group that is responsible for a Work Item to discuss what deliverables that will be produced, how these are to be packaged (in Enabler or Reference Releases) and to what extent there is a need for holding a consistency review.
Consistency Reviews are held to help determine the suitability of Release packages being advanced to the Candidate state. The Consistency Review is intended to address the full range of concerns that may be raised regarding the quality and suitability of the material to be covered. To be successful, the Consistency Review will have participation by delegates that cover the full range of interests in OMA to help assure complete coverage.
The Release Planning and Management Committee of the Technical Plenary (REL) is tasked with managing the Consistency Reviews. This does not mean that participants to the actual reviews need be regular participants in this committee. It merely means that REL is expected to help with the logistics of the reviews, as well as the planning of the activities up to the review. The actual Consistency Group, as described in the Process Document, is a virtual collection of OMA delegates who support the Consistency Reviews.
This procedure description covers several aspects related to Consistency Reviews. The specific material covered is:
The key part of the preparation for the Consistency Review involves getting the material elements of the review collected and made available. This involves construction of the Enabler Release Package (ERP) or Reference Release Package (RRP) and associated files. These packages are to contain the Releases and previous review reports coming from the review of the material that is part of the Release (e.g. an RDRR for the RD, ADRR for the AD or past consistency review reports).
These additional files, as well as the ETR should be stored in the Release Package zip file for ease of distribution but it should be recognized that these files will be removed from the package before it will be published.
The ERP, which defines the package that will be published for an Enabler Release, normally includes the following files:
In addition the following document which is not part of the ERP is to be reviewed:
The ETR is to be stored in the Enabler Release Package along with the Enabler Test Review Report (ETRRR) which indicates that it has undergone a review.
Note that in case that the ERP contains documents that originate from activities performed prior to that they were handled in OMA (legacy material), exemptions may be granted by REL to allow the ERP to undergo a consistency review although no RD, AD and/or ETR documents exist.
The RRP, which defines the package that will be published for a Reference Release normally includes some or all of the following files:
Reference Release Description (RRELD) that, among other things, describes the reference release
Requirements Document (RD)
Architecture Document (AD)
White Papers (WP)
Needed Support Files (e.g. DTDs, Schema Descriptions)
There is no ETR related to a Reference Release.
Before a Consistency Review may be requested it is expected that a group will disclose to the Technical Plenary that the material is close to being completed so that interested parties can prepare for the upcoming consistency review. This disclosure should be provided during a Working Group presentation during a plenary session and should give a rough idea of when the material will be available and what kind of functionality that will be included with the release. Recognize that the actual review would occur about a month after the material is available which may be following the subsequent plenary session.
The Working Group shall then request the initiation of a consistency review using the OMA-CONSISTENCY-REVIEW mail list. Along with the request shall the link to the ERP/RRP be provided, plus the name of one or several review report editors. Optionally, the submitting group may also suggest a review date and time for the consistency review.
Upon reception of the request, REL shall also assign a moderator for the review who are responsible for continued preparations for the review, holding the review and the events following up to the point when the review is closed (by default the REL chair, but this task may be agreed to be assigned to other members of the committee).
Before being scheduled, the moderator shall perform a cursory review of the available material (e.g. make sure expected files are present in the Reference Release) and if there are any problems, a quick response to the submitting party noting the faults found so that corrective actions may be taken.
With a proper review package available, the start of the consistency review is announced and a date for the review shall be negotiated. Along with the initiation of the review the moderator shall also assign a prefix that is to be used in the subject line of emails containing any dialogue related to the consistency review. The review date should be agreeable to both the submitting group (able to participate), REL and members that indicate an interest to participate in the review. The review date shall be negotiated via the OMA-CONSISTENCY-REVIEW mail list with at least a two working days deadline for people to react to the time that is suggested.
The length of the review period (the time between when the review is initiated and the review meeting) is dependent on the contents of the Release. Typically, the review period for a Release Package undergoing a first review would be a minimum of 14 days in order to provide enough time for members to review, collect and submit inputs to the review. The length of the review period is also dependent on to what extent the contents of the Release Package already has undergone review. In some cases, it may even be suggested that no consistency review will be needed. This too is to be negotiated using the same procedure as for agreeing on a review date.
When determining the length of the review period, Cconsideration should also be given to the amount of material as well as the amount of other material already undergoing consistency reviews. Extra time may be warranted for a large enabler releases or in cases when several other consistency reviews already are ongoing and a shorter interval may be acceptable for a simple revision (follow-up review) when there is little other consistency review activity.
Once the meeting time is agreed, the moderator shall upload a meeting agenda to the REL portal as well as create a calendar item for the review meeting (also on the REL portal).
Several activities need to be performed in support of the consistency review. These checks can be performed somewhat independent of any face-to-face session as they are primarily not dependent upon interpretation of the specifications. The latest version of the Consistency review guidelines document (which can be found on the Portal) lists the checks that are expected to be performed on all specifications.
In particular it should be noted that since the Requirement Document normally already has been approved as a Candidate by the Technical Plenary, any checks of this document as part of the consistency review are expected to be done to ensure that the other material matches its content. In case of needed changes to the RD, normal CR handling for approved Candidate specifications is expected, including demotion of the RD in case that major changes are made to it (class 0 or 1 CRs). Class 0 changes (new functionality) are not expected to be introduced at this late stage.
All comments submitted to the consistency review must be submitted using the consistency review mail list and with the announced prefix in the subject line.
It is expected that the contributors of review comments ease the work of the review report editor by providing comments in the same format that is used in the consistency review report. It is also strongly recommended that these contributors suggest resolutions to these review comments, in case they have a view on how a matter can be resolved. It is also recommended that the contributors clearly distinguish between editorial comments and more substantiate ones.
Although it is allowed to submit comments directly via email, it is recommended to use input contributions that are uploaded to the REL portal in case the number of comments is not small. This is especially important for late comments to the review, as it then may not be possible for the review report editor to be able to incorporate these into the review report in time for the review.
It is important that a single report be produced and available for delegates to be able to consider all issues that may be raised against a release. The review report editor is responsible for collecting the various comments into a single report document. The editor should preferably be from the submitting group. The review report shall be a permanent document which identity is allocated from the REL PD area.
There are two main pieces of information for each issue collected in a
review: the description of the issue and the response from the submitting group.
The issue description should be a clear statement of the issue with reference to
the document and section where the issue is raised. In addition, it should note
the source of the issue (e.g. email from X, WG Y, review meeting agreement,
etc). The response from the submitting group should clarify how the submitting
group has chosen to address the raised issue.
The working group that is responsible for the release may start work on the
resolutions to the consistency review comments prior to that the consistency
review is being held. They may also request clarification from reviewers on
comments that they do not understand. When working with resolutions to review
comments prior to the review the group shall however bear in mind that these
comments have not yet been agreed, so they could potentially be changed or not
agreed during the review. The review report editor may already at this point
start noting the resolutions towards the review comments in the consistency
review report, but these are not to be discussed during the consistency review
meeting.
The Release Package that is submitted for consistency review
should be frozen during the time period up to that the review is being held to
ensure that all reviewers are reviewing the exact same material. The reviewers
shall not consider any changes that are made during this time
period.
The formal Consistency Review should be a live session (face-to-face or teleconference) where interested parties may actually discuss issues and determine if they are relevant.
The Review report editor should capture the review comments in the consistency review report and upload a revision of the report to the PD area of the REL portal at the latest the day before the review is held. Late input received after the report has been created will be handled separately after the review report has been gone through. During the review, there should be determination on whether the issues raised so far are correct and it may then be determined that the issue should be changed or removed.
New issues may also be raised during the review. Following any discussion, there should be a determination of whether the issue will be captured in the review report. The editor will collect any such issues.
Care should be taken in removing comments from people not participating in the review; in those cases it would be better to note the group view in the response to avoid losing issues from the report.
The moderator of the consistency review is responsible for taking minutes (this task may be delegated to another review participant).
After the Consistency Review session, the review report editor should (in case changes have been made) upload a new revision of the review report to the PD area for REL and announce its availability by sending a mail to the OMA-CONSISTENCY REVIEW mail list..
The moderator of the review shall prepare the minutes of meeting from the consistency review and then announce a deadline for examination of the review report and the minutes of meeting.A minimum of two working days shall be allowed for this.
The participants should consider whether all of the issues raised have been
captured correctly and may seek revision if an issue is missing or
mischaracterized. The agreement of whether the report is correct may take place
with use of email and does not require any physical meeting.
After the
minutes and review report have been agreed, the moderator shall announce this on
the consistency review mail list.
The group submitting the release for review is then responsible for resolving and responding to the issues that were raised. The review report response area should be filled in for all issues. Responses may be of several forms. These may include:
When the review report responses are finished, the review report editor should upload the updated report as a PD to the REL portal and send a mail to the OMA-CONSISTENCY-REVIEW mail list. In addition, if any changes were required in the Release Package (e.g. spec updates), it should be revised and made available as a PD on the REL portal as well. Note that the consistency review report also should be updated to indicate if any additional changes (not caused by the review comments) have been made to the release.
The submitting group may request a follow-up review if there were issues that needed further clarification. Such a follow-up would nominally be handled via email but another live meeting could be used, as needed.
Once the final version of the review report is submitted, a cursory review should be performed to make sure that the changes outlined in the responses have been provided in the revised Release Package. This can typically be handled over email and a minimum period of 2 working days should be allocated to allow review participants to validate that the responses and changes are satisfactory. This may lead to further updates of the review report, as well as changes to the Release Package. Once this has been done, the review can be considered completed.
Note that if the updated Release Package contains changes that are not originating from comments raised during the original consistency review, then the reviewers may request that a follow-up consistency review is scheduled to specifically allow for review of these changes. The changes should be substantial enough to motivate such a follow-up review, either by number or impact on specifications. The request for a follow-up review is to occur during the time period during which the cursory review of the material is taking place. The moderator of the review will manage such requests and determine whether there is grounds for holding another review and will also suggest a time period for the review which will be negotiated in the same way as for the initial review.
There is no ‘Approval’ granted by completing the review. It merely signifies that there are responses for all of the issues raised and that the changes indicated have been performed. It should however be noted that a prerequisite for bringing a Release to the Technical Plenary for approval is that the Release Package is complete, meaning that:
a) all planned requirements, as defined in the RD with agreed updates post RD approval have been addressed,
b) all necessary aspects of architecture, security and the function have been specified,
c) for Enabler Releases that any interoperability requirements at the specification level is complete, including the Enabler Test Requirements
d) the documents have no known omissions or problems.
e) there are no other known substantive issues outstanding.
When the working group has determined that all of this has been achieved, the moderator of the consistency review will announce to the consistency review mail list that the consistency review is completed. Actions shall then be taken by REL and the working group owning the release to submit the release for Candidate approval.
If there are disagreements with the results of the Consistency Review,
members are entitled to raise their objection when the material is brought to TP
for consideration as a Candidate Release.
A graphical representation of this flow can be found here.